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METHOD FOR GENERATPJG CUSTOMIZED PHOTO ALBUM PAGES 
AND PRINTS BASED ON PEOPLE AND GENDER PROFILES 

FIELD OF THE INVENTION 

5 The invention relates generally to the field of digital image 

processing, and in particular to a method for suggesting suitable designs for album 
pages and customized prints based on the image content and the demographics of 
people present in the images. 

1 0 BACKGROUND OF THE INVENTION 

With the advent of digital imaging technology, it has become 
possible for consumers to create customized products such as photo greeting 
cards, framed prints and albums inexpensively. Even if the original image is 
available only as a traditional paper print, it is possible to use a photo kiosk to 

1 5 scan the image and use it creatively in a new print product. 

There are a number of commercially available software products 
that provide custom printing services to be used for printing on a home printer. 
Photo kiosks provide similar services for printing at the kiosk itself. Typically, 
these products provide a very wide selection of possible colors and designs for use 

20 as background and/or around the border of the print. Most products also allow for 
the addition of text on or aroimd the image, and there are numerous choices of 
fonts, sizes and colors available for this pxirpose. The user selects an image or a 
set of images to be used. Examples of custom printing software include 
Microsoft's Picture It, ArcSofl's PhotoPrinter, and others such as PrintSix, 

25 PhotoElf and ImageBuddy. 

Albuming software products provide a variety of templates for page 
layout and themed decorations, in addition to backgrounds and borders. Some 
albuming softwares offer choices of music or audio to include with each album 
page of a digital album (soft copy), to produce a multimedia presentation for the 

30 viewer. Some examples of albviming softwares include DogByte's Creative Photo 
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Album, flipalbum.com's FlipAlbum, Xequte's Diji album and also Microsoft's 
Picture It. 

In a typical situation, the consumer needs to spend a lot of time at a 
task such as albuming or generating customized prints, most of the time being 
5 spent in searching for suitable image sets from the consumer's image collection 
and trying out the various options and their combinations till a visually pleasing 
combination is found. Since software products in this application area typically 
offer hundreds of options for each choice to be made (in some cases, thousands), it 
is increasingly difficult for a consumer to produce an effective presentation of 

10 their image(s) easily and quickly. The consumer can get tired of exploring all 
possible combinations and settle on one which does not meet all their criteria, 
causing dissatisfaction. 

In U.S. Patent No. 6,389,181 issued on May 14, 2002, Shaffer et al 
discuss a method for automatically producing a photocoUage by employing image 

1 5 recognition techniques. However, their method of selecting layout and page 

design is based on the availability of a customer profile containing the customer's 
color preferences, layout preferences and design considerations. The image 
content is not used in this process. Face recognition is used for grouping images, 
assuming that sample faces and identification of all members of the customer's 

20 family are available in the customer profile. A scenario where a comprehensive 
customer profile such as this is available is hard to achieve in practice. 

In the Intemational Application WO 02/01537 A2 published on 
January 3, 2002, Cohen-Solal et al use demographic information and the number 
of people, as in the present invention, for adapting the content of a display screen. 

25 Here, these characteristics are estimated from live video feed and mapped to 

audio- visual presentations that will be most relevant for viewing, given the current 
audience. The mapping is customized to cater to the interests of specific groups, 
for example, a primarily male audience may prefer sports highlights or a 
frequently moving audience may require short presentations. A similar automatic 

30 customization would be usefiil but is currently unavailable for the album 
generation process. 
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A need therefore exists to provide the consumer with fewer, but 
well-crafted, choices that would help them to achieve the given task (albuming, 
custom printing, etc.) quickly and satisfactorily. The suggested choices should be 
appropriate for the image content displayed, and provide a complete solution that 
5 takes into account all the interactions between individual elements, e.g., the color 
of backgroimd should work well with the border chosen. This process would not 
limit the consumers' choices, as the suggestions could provide the starting points 
for more explorations if the consumer so wishes. 

10 SUMMARY OF THE INVENTION 

The present invention is directed to overcoming one or more of the 
problems set forth above. Briefly summarized, according to one aspect of the 
present invention, the invention resides in a method for generating customized 
photo album pages for a collection of digital images, where the method comprises 

15 the steps of: (a) obtaining a plurality of digital images; (b) automatically grouping 
the images into one or more image sets based on at least one of grouping by events 
and grouping by people present in the images; (c) automatically profiling the 
images within each image set according to image content, thereby relating a 
particular image set to one or more design themes; (d) providing a database of 

20 design elements that may be used to generate customized photo album pages; (e) 
utilizing the design themes generated by the profiling step to automatically suggest 
one or more design elements fi-om the database of design elements; and (f) 
automatically generating a customized photo album page layout for each image set 
by applying one or more of the suggested design elements to the page layout. 

25 The manual selection of image sets for albuming/custom printing is 

a time-consuming process involving browsing large image collections. According 
to one of its advantages, this invention provides an automated solution to the 
problem of finding image sets that form a cohesive unit for the albuming task. 
The user can select image sets by event, or by people present in the images, or a 

30 combination of the two methods, e.g., images of a particular person taken during a 
particular event. 
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It is a well-known observation that the type of themes selected by 
consumers for displaying their image(s) is closely related to the gender and ages of 
the people appearing in the images. For example, a baby's picture is displayed 
with themed embellishments such as toys. A girl's portrait may be displayed with 
5 colors such as pink and floral motifs. An adult male and female may fit the model 
of a couple, and the consumer may be looking for themes suitable for a couple. 
Similarly, a scene without any people (such as scenery) may need to be displayed 
with colors that complement the color composition of the scene. 

According to another of its advantages, the present invention uses 

10 the demographics, image type (portrait, large group, etc.) and the color content of 
images as the basis for selection of a group of suggested display elements. The 
options suggested are based on a custom-designed mapping between image 
characteristics and display element groups that takes into account current trends in 
design and cultural preferences. Once created, this mapping would produce 

1 5 suitable photo decoration options for different types of situations, making the 
process of photo product generation at the kiosk or home more pleasurable and 
less time consuming. 

These and other aspects, objects, features and advantages of the 
present invention will be more clearly understood and appreciated from a review 

20 of the following detailed description of the preferred embodiments and appended 
claims, and by reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a flowchart of the overall system for automation-aided 
25 albuming proposed in the present invention. 

Figure 2 is a flowchart of the method of determining display 
elements in the present invention. 

Figure 3 is a flowchart of the gender classification method 
according to the present invention. 
30 Figure 4 is an illustration of the location of facial feature points on 

a human face. 
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Figure 5 is a table of measurement-based features used for gender 
classification in the present invention. 

Figure 6 A defines a set of bounding boxes identifying a set of 
image patches and Figure 6B shows a table of histogram features based on 
5 differences between the patches identified in Figure 6A, and as used for gender 
classification in the present invention. 

Figure 7 is an example of a table mapping photo profile 
information to design groups. 

Figure 8 shows two examples of design elements suggested by the 
10 proposed system. 



DETAILED DESCRIPTION OF THE INVENTION 

Because image processing systems employing albimiing software 
are well known, the present description will be directed in particular to attributes 

15 forming part of, or cooperating more directly with, method in accordance with the 
present invention. Attributes of the method not specifically shown or described 
herein may be selected fi-om those known in the art. In the following description, 
a preferred embodiment of the present invention would ordinarily be implemented 
as a software program, although those skilled in the art will readily recognize that 

20 the equivalent of such software may also be constructed in hardware. Given the 
method as described according to the invention in the following materials, 
software not specifically shown, suggested or described herein that is useful for 
implementation of the invention is conventional and within the ordinary skill in 
such arts. If the invention is implemented as a computer program, the program 

25 may be stored in conventional computer readable storage medium, which may 

comprise, for example; magnetic storage media such as a magnetic disk (such as a 
floppy disk or a hard drive) or magnetic tape; optical storage media such as an 
optical disc, optical tape, or machine readable bar code; solid state electronic 
storage devices such as random access memory (RAM), or read only memory 

30 (ROM); or any other physical device or medium employed to store a computer 
program. 
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The present invention provides the consximer with a suggested set 
of choices for display elements that are appropriate for the type(s) of picture(s) 
used in the photo product. Image sets can be selected based on grouping by events 
or by people or a combination of the two. Display elements consist of image 
5 backgrounds, borders, frames, layouts, background music (for digital albums) and 
any other display options provided by the system. The selection of the various 
display elements is based on the content of the image; with emphasis on the 
description of people present in the image. Automatic image processing 
algorithms are used to determine if people are present in an image, and 

10 demographic information (gender and age) about them. This invention provides 
an automated solution to the problem of finding image sets that form a cohesive 
imit for the albuming task. The user can select image sets by event, or by people 
present in the images, or a combination of the two methods, e.g., images of a 
particular person taken during a particular event. 

15 Referring to Figure 1, the consumers' digital image collection or a 

subset thereof 10 is grouped by event 12 and by people present in images 14, The 
consumer may choose either criteria — event or people — for grouping, or both in 
any order, based on the semantics of the task. For example, to create an album of 
the events in a baby's first year, the user may first group images by people, and 

20 then group by events after selecting the group depicting the baby; whereas, to 
create an album to share vacation experiences with a fiiend, the user may first 
group by event (vacation) and then by people (to find images containing the user). 
Grouping by event 12 is achieved using the method described in U.S. Patent No. 
6,351,556, entitled "A Method for Automatically Comparing Content of hnages 

25 for Classification into Events", by Loui and Pavie issued on February 26, 2002; 

and grouping by the identity of people present in the images 14 uses the method of 
clustering by facial similarity described in U.S. Serial No. 10/143,272, entitled 
"Method and Apparatus for Organizing and Retrieving Images Containing Human 
Faces" filed 1 0 May 2002 in the names of Chen et al, both of which are 

30 incorporated herein by reference. 
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The user can then select the group(s) 16 of images to include in the 
album. The grouping algorithms also provide emphasis scores to each image 
relative to other images in the group. In grouping by people, an emphasis score is 
assigned to each retrieved image, given by the fraction of the image covered by the 
5 faces of the specified people. In event-based grouping, this score is based on 
image quality (sharpness, contrast, etc.) and composition. The emphasis scores 
are used by the layout engine 18 to generate page layouts, as described in U.S. 
Serial No. 09/750,858, entitled "System and Method for Automatic Layout of 
Images in Digital Albums" by Geigel and Loui filed on 29 December 2000, which 
10 is incorporated herein by reference. The design engine 20, which uses the method 
described in connection with Figure 2, is then used to suggest designs appropriate 
for the selected image set from the database of design elements 22 based on the 
photo profile_generated. The album page(s) 24 are created with the selected layout 
and design choices. 

15 Referring to Figure 2, which shows the design engine 20 of Figure 

1, human faces are located 110 in the digital image(s) selected by the user for 
creating a customized print or album page. There are a number of known face 
detection algorithms that can be used for this purpose. In a preferred embodiment, 
the face detector described in commonly-assigned U.S. Serial No. 10/042,605, 

20 entitled "Method and System for Processing Images for Themed Imaging 

Services" filed January 9, 2002 by Nicponski is used, which is incorporated herein 
by reference. The face detector is based on a Bayesian classifier trained with 
examples of face and non-face images. The face detector outputs the locations 
and sizes of faces found in the image(s). Based on the number and size of faces 

25 detected, the image is assigned 112 an image type - for example, portrait (or 

close-up), group portrait, snapshot, large group or non-face. Each face detected is 
classified 114 as baby, child, adult or senior. A method for assigning a face to an 
age category is described in U.S. Patent No. 5,781,650, entitled "Automatic 
Feature Detection and Age Classification of Human Faces in Digital Images" by 

30 Lobo issued on July 14, 1998, which is incorporated herein by reference. That 
method uses ratios of facial measurements that are known to change with age due 
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to bone structure development. The adult faces are further classified 116 as male 
or female. A gender classification method described in "Learning Gender with 
Support Faces" by B. Moghaddam and M.-H. Yang in IEEE Transactions on 
Pattern Analysis and Machine Intelligence^ vol. 24, no. 5, pp. 707-71 1, May 2002, 
5 can be used for the purpose, which is incorporated herein by reference. However, 
this method is not accurate (a correct classification rate of about 70% is observed) 
on un-aligned faces, where the eye locations are not known accurately (as in this 
case), hi the preferred embodiment, gender classification involves the steps shown 
in Figure 3. 

1 0 The approximate eye locations are obtained 120 from the face 

detector and used to initialize the starting face position for facial feature finding. 
Eighty-two facial feature points are detected 121 using the Active Shape Model- 
based method described in "An automatic facial feature finding system for portrait 
images", by Bolin and Chen in the Proceedings oflS&TPICS conference, 2002, 

1 5 which is incorporated herein by reference. Their method uses local texture 

matching to localize feature points, followed by the application of global shape 
constraints leamed from training examples. Figure 4 shows the locations of the 
feature points on a face. 

Some facial measurements that are known to be statistically 

20 different between men and women (ref. "Anthropometry of the Head and Face" by 
Farkas (Ed.), 2""* edition. Raven Press, New York, 1994, and "What's the 
difference between men and women? Evidence from facial measurement" by 
Burton, Bruce and Dench, Perception, vol, 22, pp. 153-176, 1993) are computed 
122. Figure 5 lists the features used, where P« refers to facial point number n 

25 from Figure 4. The features are normalized by the inter-ocular distance, to 

eliminate the effect of differences in the raw size of the face. For symmetrical 
features, measurements from the left and right side of the faces are averaged to 
produce more robust measurements. 

According to the invention, additional performance improvement is 

30 obtained by using the presence or absence of hair in specific locations on and 

around the face as cues for gender determination. These features are incorporated 
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124 as a difference in gray-scale histograms between the patch where hair may be 
present, and a reference patch on the cheek that is typically hairless. The 
bovmding box for the patches are automatically computed as shown in Figure 6A 
based on the feature points, and relevant histogram difference-based features are 
5 computed as shown in Figure 6B. 

Binary classifiers are constructed 126 using each of the twenty-two 
single features (listed in Figure 5 and 6) separately. Simple Bayesian classifiers 
described in standard literature (Pattern Classification by R.O. Duda, P.E. Hart 
and D.G. Stork, John Wiley and Sons, 2001) are trained on large sets of example 

1 0 male and female faces to produce the single feature-based binary classifiers. The 
classification accuracy of each of these binary classifiers ranged fi^om 55 to 75%. 

The binary classifiers were combined using the AdaBoost 
algorithm to produce 128 an improved final classifier. AdaBoost is a well-known 
algoridmi for boosting classifier accuracy by combining the outputs of weak 

15 classifiers (such as the single feature binary classifiers described above). The 
weighted sum of outputs of the weak classifiers is compared with a threshold 
computed automatically fi-om the training examples. A description and 
application of this method is available in "Rapid Object Detection using a Boosted 
Cascade of Simple Features" by P. Viola and M. Jones, in International 

20 Conference on Computer Vision and Pattern Recognition^ 2001, which is 

incorporated herein by reference. The classification accuracy of the final classifier 
obtained using AdaBoost was 90% on un-aligned faces. 

Referring back to Figure 2, and based on the information computed 
above, each image is assigned 118 a photo profile, which includes the image type, 

25 the ages and genders of the people (if any) present in the image, and the color 

distributions in the image. The color distribution is described as the proportion of 
pixels in each of the major color categories as defined in "The ISCC-NBS method 
of designating colors and a dictionary of color names" by K.L. Kelly and D.B. 
Judd in the National Bureau of Standards Circular (553)^ Nov 1 , 1 955 - red, blue, 

30 yellow, green, pink, purple, brown, orange, olive, white, black and gray. 
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A large number of consumer images from family photo collections, 
where the relationships between individuals were known, are analyzed to 
detemiine if there are any likely relationships given the age and genders detected. 
These common scenarios are used to pick out designs 119 appropriate in that 
5 scenario. Figure 7 shows some examples of tables that map the photo profiles to 
design themes. Each design group needs to be custom-designed and updated 
periodically to reflect current trends in design and cultural preferences. The color 
of the design elements, border and background are selected based on the color 
distribution in the image. The color scheme in each case may be coordinating 

10 (colors pre-dominant in the image are chosen), complementary (colors contrasting 
with the pre-dominant colors in the image are chosen) or neutral (unsaturated 
versions of the pre-dominant colors are chosen). If the product being created is a 
multimedia presentation, music clippings are also suggested as part of the design 
theme based on gender and ages of people in the images. 

15 At the end of the process, the user is presented with a limited 

number (typically around 5) of complete design selections. Abstracts and neutrals 
are included in the options in all cases, since there may be errors in face detection 
and the age/gender determination steps. Figure 8 shows two examples of the 
design elements suggested for a page with portrait pictures of a baby (left) and a 

20 page with group pictures of girl children (right). 

The invention has been described with reference to a preferred 
embodiment. However, it will be appreciated that variations and modifications 
can be effected by a person of ordinary skill in the art without departing from the 
scope of the invention. 



25 
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locate faces step 
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assign image type step 
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classify by age group step 
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classify by gender step 
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create photo profile step 
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choose display elements step 
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compute facial measurements step 
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compute histogram-based features step 
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compute binary classifiers step 
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combine binary classifiers step 



